New Second-Order Algorithms for Recurrent Neural Networks Based on Conjugate Gradient

In this paper we derive two second-order algorithms, based on conjugate gradient, for on-line training of recurrent neural networks. These azgorithms use two different techniques to extract second-order information on the Hessian matrix without calculating or storing it and without making numericaz approximations. Several simulation results for non-linear system identzjication tests by locally’ recurrent neuraZ networks are reported for both the off-line and on-line case, comparing with firstorder algorithms and showing faster training. The Conjugate Gradient (CG) algorithm is based on the following two iterative formulas respectively for updating weights wk and conjugate directions pA[9]: